Field of the Invention
The present invention is a method and system to recognize which product a shopper intends to find or purchase based on the shopper's trajectory in a retail aisle.
Background of the Invention
The current consumer and market-oriented economy places a great deal of importance on shoppers' behavior and decision making process in a retail space—how shoppers look for and interact with products and how they make purchase decisions while shopping in stores. There is also a consensus within the market research community that today's consumers make most of their purchase decisions in stores. Until shoppers finally make decisions to purchase certain products, they often make comparisons with alternative products and try to gather more information about them. Even when they don't purchase some of the items with which they interacted, the information about the interactions themselves—which products they were interested in and how much time they spent with the products—constitutes very valuable marketing data. Transaction data alone is not able to provide such information.
Consumer interaction with products occurs in retail spaces at an immeasurably high number and frequency. Consumer behavior often changes according to time of the day, season, and economic environment. It would, therefore, take much resource to measure the behaviors and generate useful data in order to derive meaningful conclusions, despite these variables. It is the main goal of the present invention to collect and analyze consumer behavior data captured in a retail space using videos. The current state-of-the-art video analytic technology has not advanced enough to recognize very fine detail of human behavior; however, simply measuring how shoppers move around in a retail space has been quite successful due to advances in human tracking technology. Such technology is able to generate a huge amount of consumer trajectory by automatically analyzing videos captured by in-store cameras. The present invention specifically concerns the exploitation of such trajectory data to extract shopper behavior information.
In a typical shopping scenario, a shopper browses through retail aisles with an intention to buy certain products or to look for any desirable products. Then she/he notices a product or a product category that catches her/his attention (regardless of whether it was intended or not), approaches the shelf, interacts with products, and makes a decision as to which one to buy or not to buy at all. If the shopper becomes interested, she/he engages with the product or category by approaching and stopping at the shelf. Then she/he directly interacts with the intended product or further explores different options within the category or other categories. The interaction involves checking the price, reading the labels, placing the item in the shopping cart, or returning the item to the shelf. The shopper then makes a decision to pick up the product, and continues to browse for other products. Alternatively, the shopper can return the product to the shelf and look for other products in the same categories. It is one of the goals of the present invention to recognize such behaviors based on the shopper's trajectory.
Recent developments in computer vision and artificial intelligence technology make it possible to detect and track people from video sequences so that their trajectories can be estimated. More specifically, body detection can locate any human body images from a video sequence and individually keep track of their movements, so that the system can estimate each shopper's positions over time.
The estimated trajectory contains much information about the shopper's intention at each instance. More specifically, various features—such as position, speed, and orientation of the movement—reveal the shopper's current state. For example, a fast movement to the same direction of the aisle usually tells that the shopper is walking toward a section and has not yet reached the point to look for a particular product. Slower walking reveals that the shopper is in or very near the intended section and is looking for an intended product. Very slow movement or stopping near one of the (left or right) shelves may reveal that the shopper is interacting with a product on that shelf or is standing back from the intended shelf to have a wider view. If the shopper has paused to look for a product and finally finds a product, she/he approaches the shelf to the direction close to the lateral direction. Typical shopping scenarios involve two or more of these behaviors; the progression of the behaviors follows certain patterns.
The present invention provides a method to interpret each shopper's trajectory to make an inference of what the shopper's intention is at the moment. Based on the behavior model mentioned above, the method of the invention interprets the shopping trajectory into such atomic behaviors to further determine the shopper's intention. In one of the exemplary embodiments, the invention adopts a rule-based approach to interpret the shopping trajectories. In another exemplary embodiment, the invention employs a probabilistic graphical model—such as a Hidden Markov Model—to interpret the shopping trajectories.
There have been prior attempts for tracking people's motion for the purpose of understanding their behaviors.
U.S. Pat. Appl. Pub. No. 2002/0085092 of Choi, et al. (hereinafter Choi) disclosed a method for modeling an activity of a human body using the optical flow vector from a video and probability distribution of the feature vectors from the optical flow vector. Choi modeled a plurality of states using the probability distribution of the feature vectors and expressed the activity based on the state transition.
U.S. Pat. Appl. Pub. No. 2003/0053659 of Pavlidis, et al. (hereinafter Pavlidis) disclosed a method for moving object assessment, including an object path of one or more moving objects in a search area, using a plurality of imaging devices and segmentation by background subtraction.
U.S. Pat. Appl. Pub. No. 2004/0120581 of Ozer, et al. (hereinafter Ozer) disclosed a method for identifying the activity of customers for marketing purposes or the activity of objects in a surveillance area, by comparing the detected objects with the graphs from a database. Ozer tracked the movement of different object parts and combined them to high-level activity semantics, using several Hidden Markov Models (HMMs) and a distance classifier.
U.S. Pat. Appl. Pub. No. 2004/0131254 of Liang, et al. (hereinafter Liang) also disclosed the Hidden Markov Models (HMMs) as a way, along with the rule-based label analysis and the token parsing procedure, to characterize behavior. Liang disclosed a method for monitoring and classifying actions of various objects in a video, using background subtraction for object detection and tracking.
U.S. Pat. Appl. Pub. No. 2008/0018738 of Lipton, et al. (hereinafter Lipton) disclosed a video analytics engine and an activity inference engine to generate video primitives and determine whether an activity of interest occurred in a video using the video primitives.
Pavlidis was primarily related to monitoring a search area for surveillance using visual tracking. Choi, Ozer, Liang, and Lipton all define behavior primitives and try to determine whether a given video stream contains the sequence of these primitive behaviors using probabilistic graphical model approaches such as HMM. The present invention employs a similar approach to model the shoppers' behaviors in retail aisles to determine their intentions, but it only requires trajectory data to model and infer the behaviors; it does not need additional visual cues as required in the prior inventions.
There have been prior attempts for tracking customers or measuring customer interactions with products using communication devices for the purpose of understanding customer behaviors.
U.S. Pat. No. 6,659,344 of Otto, et al. (hereinafter Otto) presents a shopper behavior monitoring system using RFID tags attached to products and RFID scanners installed in shopping baskets, so that the system can detect product purchase at the shelf and identify the purchase items. In U.S. Pat. No. 7,006,982 of Sorensen (hereinafter Sorensen) and U.S. Pat. Appl. Pub. No. 2008/0042836 of Christopher (hereinafter Christopher), a shopper is tracked using a wireless tracking device installed in the shopping cart. The trajectory of the shopper is analyzed to deduce the interaction of the shopper with the products. In U.S. Pat. Appl. Pub. No. 2005/0102183 of Kelliher, et al. (hereinafter Kelliher), the comparison between the acquired items and checkout items is made based on the shopper location and behavior estimated from RFID tracking devices, so that potential fraud can be detected. In U.S. Pat. Appl. Pub. No. 2007/0067221 of Godsey, et al. (hereinafter Godsey), the RFID system is used to detect product purchase. In all of these inventions, special devices are installed on shopping baskets/carts and/or attached to products to measure the shopping and purchase behavior of customers. The present invention can also process trajectory data generated from such devices. However, it simply requires the shoppers' trajectory data, and not the data regarding the product displacement. In its primary exemplary embodiment, the present invention utilizes video cameras to track shoppers and generate trajectories without using any costly and cumbersome devices. Unlike some of the mentioned prior inventions, the present invention does not depend on the movement of the carts—which limits the scope of the application—to track the customer behavior.
In summary, the present invention provides an approach to analyzing a shopper's trajectory in retail aisles to derive the shopper's intention. Unlike some of the prior inventions, the present invention does not require a dedicated communication device to track shoppers' movements or to recognize the displacement of products. Unlike some of the prior inventions, the present invention does not require any visual information—such as identified body parts of the shoppers or the movements of body parts—other than the trajectories of the shoppers. As some of the prior inventions, the present invention utilizes a model of behavioral primitives and the probabilistic relations between them. However, the present invention adopts a set of dynamic behavioral primitives—such as walking, browsing, approaching, and interaction—specifically chosen to deduce which products the shopper intends to examine or buy.